Skip to content

Add Qwen 3.8 27B Megatron support - #820

Merged
FurtherAI merged 7 commits into
mainfrom
austin/qwen3_8_27b_review
Aug 28, 2026
Merged

Add Qwen 3.8 27B Megatron support#820
FurtherAI merged 7 commits into
mainfrom
austin/qwen3_8_27b_review

Conversation

@FurtherAI

Copy link
Copy Markdown
Collaborator

Summary

Adds qualified Megatron support for Qwen/Qwen3.8-27B by reusing the existing
Qwen 3.5/3.6 dense hybrid handler. Qwen 3.8 has the same model geometry and
weight schema, so this does not add a new handler or fork the linear-attention
implementation.

MTP remains deliberately disabled, consistent with the other Qwen 3.5-family
models in ART.

Differences from #810

This carries forward #810's small registry, TrainerRank, and focused-test
changes on current main, while resolving the minor qualification gaps found
when running the complete model-support workflow:

  • bases the change directly on current main instead of the older
    codex/require-peft-0191 branch;
  • pins the exact qualified Qwen 3.8 Hugging Face revision;
  • selects the deterministic FP32 GDN reference by handler identity rather than
    literal qwen3.5 model-name text, which also fixes the registered Qwen 3.6
    aliases;
  • records Qwen 3.8's exact measured B300 throughput fingerprint without
    replacing Qwen 3.5's calibration or performance floors;
  • constrains the root and Megatron runtime to openai<3: OpenAI 3.x uses
    httpx2 internally and rejects ART's httpx 0.x Timeout/Limits objects
    before a request reaches the socket.

Validation

The complete default handler workflow passed from clean commit
cd61066789ed331614ce9ef6b1ef136dbb345251 on 8 B300s in 12m00s. All ten
mandatory stages ran through the normal overlapped scheduler. Optional
sensitivity variants were not requested (sensitivity_skipped=true).

  • HF parity: 55/55 signals passed; output MAPE 0.004975%; worst gradient MAPE
    0.198038% (layers.2.linear_attn.dt_bias).
  • Train/inference: passed first attempt; LoRA MAPE 7.80685% against the existing
    8.05% family limit; top-20 candidate-to-target KL 0.002315 against 0.003.
  • Chat/tool template: 6/6 scenarios passed.
  • Correctness: CP1 reference and TP2/CP2/PP2/VPP2 composition passed.
  • Reduced 8-layer throughput stage: 69,435 isolated train tok/s, 64,852 E2E
    train tok/s, 4,053 accepted train tok/s, 1.0017 matched-core/isolated ratio,
    5.14% trainer underfeed, and no acceptance failures.
  • Focused registry, TrainerRank, and workflow checks: 108 passed, 8 skipped;
    the final calibration resolver check was rerun after refreshing the contract.
  • Ruff, formatting, type checking, and both lock consistency checks passed.

This supersedes the implementation proposed in #810.

@FurtherAI
FurtherAI requested a review from bradhilton as a code owner August 28, 2026 08:47
@FurtherAI
FurtherAI temporarily deployed to trainer-rank-gpu-validation August 28, 2026 08:48 — with GitHub Actions Inactive
@FurtherAI
FurtherAI merged commit aba4b85 into main Aug 28, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants